Appearance Feature Extraction versus Image Transform-Based Approach for Visual Speech Recognition

In this paper we propose a new appearance based system which consists of two stages: visual speech feature extraction and classification, followed by recognition of the extracted feature, thereby the result is a complete lip-reading system. This lip-reading system employs our Hyper Column Model (HCM) approach to extract and classify the visual features and uses the Hidden Markov Model (HMM) for recognition. This paper addresses mainly the first stage; i.e. feature extraction and classification. We investigate the HCM performance to achieve feature extraction and classification and then compare the performance when replacing HCM with Fast Discrete Cosine Transform (FDCT). Unlike FDCT, HCM could extract the entire features without any loss. Also the experiments have shown that HCM is generally better than FDCT and provides a good distribution of the phonemes in the feature space for recognition purposes. For fair comparison, two databases are exploited with three different sets of resolution for each database. One of these two databases is designed to include shifted and scaled objects. Experiments reveal that HCM is capable of recovering and dealing with such image restrictions whereas the effectiveness of FDCT drops drastically especially for new subjects.